Video acquisition devices generate massive amounts of data. Efficient use of this data is of importance for video editing, video summarization, fast visualization and many other applications related to video management and analysis.
As illustrated in Koprinskaa et al., (“Temporal video segmentation: A survey.”, Signal Processing: Image Communication, 16 (5), 477-500 (2001)), temporal video segmentation is a key step in most existing video management tools. Many different types of algorithms have been developed to perform the temporal segmentation.
Early techniques focused on cut-boundary detection or image grouping using pixel differences, histogram comparisons, edge differences, motion analysis and the like, while more recent methods such as presented in U.S. Pat. Nos. 7,783,106B2 and 8,363,960B2 have also used image similarity metrics, classification and clustering to achieve the same goal.
In some applications as the ones in Sun, Z. et al. (“Removal of non-informative frames for wireless capsule endoscopy video segmentation”, Proc. ICAL pp. 294-299 (2012)) and Oh, J.-H. et al. (“Informative frame classification for endoscopy video”, Medical Image Analysis , 11 (2), 110-127 (2007)), the problem of temporal video segmentation may be reformulated as a classification problem that distinguishes between informative and noise images.
In US20070245242A1, temporal video segmentation has been coupled with the computation of similarity across scenes so as to produce video summaries.
In the medical device area, and in particular in the field of endoscopy, evaluation of motion patterns has played an important role in the analysis of long videos.
In U.S. Pat. No. 7200253B2, a system to evaluate the motion of an ingestible imaging capsule and to display the motion information against time is disclosed.
Similar motion information was used in US20100194869A1 for temporal video segmentation of endoscopy videos. Fast screening of the content of the video is implemented by only displaying the first image of each temporal segment; therefore skipping all other images.
To address the same goal of fast video screening in endoscopy but without skipping images, US20100194869A1 rely on motion evaluation to compute a replay speed inversely proportional to the estimated motion.
By relying on video mosaicing tools, an efficient representation of endomicroscopic videos in which consecutive images have overlap is disclosed in U.S. Pat. No. 8,218,901B2.
To ease the interpretation of entire endomicroscopic videos, André, B. et al. (“A Smart Atlas for Endomicroscopy using Automated Video Retrieval”, Medical Image Analysis, 15 (4), 460-476 (2011)) proposed a method relying on visual similarity between a current video and videos from an external database to display visually similar but annotated cases in relation to the current video.
A similar approach is disclosed in André, B. et al. (“Learning Semantic and Visual Similarity for Endomicroscopy Video Retrieval”, IEEE Transactions on Medical Imaging, 31 (6), 1276-1288 (2012)) to complement visual similarity with semantic information. On a related topic (André, B. et al. “An image retrieval approach to setup difficulty levels in training systems for endomicroscopy diagnosis”, MICCAI (pp. 480-487). Beijing: LNCS (2010)) presented a means of evaluating a difficulty level associated with the interpretation of a given endomicroscopy video.
In clinical scenarios, video analysis may need to be performed during the procedure. To work around the issue of computational time (US20110274325A1) discloses a method that takes advantage of a freezed buffer of consecutive images to perform computationally intensive tasks while continuing the image acquisition.
As illustrated in the aforementioned work, prior art shows that a real need exists for efficient use of videos acquired with a medical device. Although efficient use of video data has been addressed both in clinical and non-clinical scenarios, none of the previous approaches teach a method to characterize the interpretability of the images composing a video acquired with a medical device.
Patent document US2006/293558 discloses a method for automated measurement of metrics reflecting the quality of a colonoscopic procedure. Patent document US2011/301447 discloses a method for classifying and annotating clinical features in medical video by applying a probabilistic analysis of intra-frame or inter-frame relationships in both spatially and temporally neighboring portions of video frames.